{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "nmt_tutorial.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lvapeab/nmt-keras/blob/master/examples/tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tNxDp3Me4zXx"
      },
      "source": [
        "\n",
        "# NMT-Keras tutorial\n",
        "---\n",
        "\n",
        "This notebook describes, step by step, how to build a neural machine translation model with NMT-Keras. The tutorial is organized in different sections:\n",
        "\n",
        "\n",
        "1. Create a Dataset instance, in order to properly manage the data. \n",
        "2. Create and train the Neural Translation Model in the training data.\n",
        "3. Apply the trained model on new (unseen) data.\n",
        "\n",
        "All these steps are automatically run by the toolkit. But, to learn and understand the full process, it is didactic to follow this tutorial.\n",
        "\n",
        "\n",
        "So, let's start installing the toolkit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0mtJWLes5JO7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "c4d6a551-9d43-45d2-a9d7-8962528f5abf"
      },
      "source": [
        "!pip install update pip\n",
        "!pip uninstall -y kapre keras tensorflow-probability albumentations datascience h5py keras-nightly # Avoid crashes with pre-installed packages \n",
        "!git clone https://github.com/lvapeab/nmt-keras\n",
        "import os\n",
        "os.chdir('nmt-keras')\n",
        "!pip install -e .\n"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting update\n",
            "  Downloading update-0.0.1-py2.py3-none-any.whl (2.9 kB)\n",
            "Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (21.1.3)\n",
            "Collecting style==1.1.0\n",
            "  Downloading style-1.1.0-py2.py3-none-any.whl (6.4 kB)\n",
            "Installing collected packages: style, update\n",
            "Successfully installed style-1.1.0 update-0.0.1\n",
            "Found existing installation: kapre 0.3.5\n",
            "Uninstalling kapre-0.3.5:\n",
            "  Successfully uninstalled kapre-0.3.5\n",
            "Found existing installation: Keras 2.4.3\n",
            "Uninstalling Keras-2.4.3:\n",
            "  Successfully uninstalled Keras-2.4.3\n",
            "Found existing installation: tensorflow-probability 0.13.0\n",
            "Uninstalling tensorflow-probability-0.13.0:\n",
            "  Successfully uninstalled tensorflow-probability-0.13.0\n",
            "Found existing installation: albumentations 0.1.12\n",
            "Uninstalling albumentations-0.1.12:\n",
            "  Successfully uninstalled albumentations-0.1.12\n",
            "Found existing installation: datascience 0.10.6\n",
            "Uninstalling datascience-0.10.6:\n",
            "  Successfully uninstalled datascience-0.10.6\n",
            "Found existing installation: h5py 3.1.0\n",
            "Uninstalling h5py-3.1.0:\n",
            "  Successfully uninstalled h5py-3.1.0\n",
            "Found existing installation: keras-nightly 2.5.0.dev2021032900\n",
            "Uninstalling keras-nightly-2.5.0.dev2021032900:\n",
            "  Successfully uninstalled keras-nightly-2.5.0.dev2021032900\n",
            "Cloning into 'nmt-keras'...\n",
            "remote: Enumerating objects: 4799, done.\u001b[K\n",
            "remote: Counting objects: 100% (69/69), done.\u001b[K\n",
            "remote: Compressing objects: 100% (55/55), done.\u001b[K\n",
            "remote: Total 4799 (delta 28), reused 30 (delta 14), pack-reused 4730\u001b[K\n",
            "Receiving objects: 100% (4799/4799), 5.75 MiB | 12.59 MiB/s, done.\n",
            "Resolving deltas: 100% (3243/3243), done.\n",
            "Obtaining file:///content/nmt-keras\n",
            "Collecting keras@ https://github.com/MarcBS/keras/archive/master.zip\n",
            "  Downloading https://github.com/MarcBS/keras/archive/master.zip\n",
            "\u001b[K     / 101.2 MB 113.8 MB/s\n",
            "\u001b[?25hRequirement already satisfied: cloudpickle in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.3.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (0.16.0)\n",
            "Collecting keras_applications\n",
            "  Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)\n",
            "\u001b[K     |████████████████████████████████| 50 kB 4.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: keras_preprocessing in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.1.2)\n",
            "Collecting h5py\n",
            "  Downloading h5py-3.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.1 MB 14.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (3.2.2)\n",
            "Collecting multimodal-keras-wrapper\n",
            "  Downloading multimodal_keras_wrapper-3.1.6-py3-none-any.whl (124 kB)\n",
            "\u001b[K     |████████████████████████████████| 124 kB 62.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.19.5)\n",
            "Requirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (0.16.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (0.22.2.post1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.15.0)\n",
            "Requirement already satisfied: tables in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (3.4.4)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.1.5)\n",
            "Collecting sacrebleu\n",
            "  Downloading sacrebleu-1.5.1-py3-none-any.whl (54 kB)\n",
            "\u001b[K     |████████████████████████████████| 54 kB 4.2 MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "  Downloading sacremoses-0.0.45-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 59.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from nmt-keras==0.6) (1.4.1)\n",
            "Collecting tensorflow<2\n",
            "  Downloading tensorflow-1.15.5-cp37-cp37m-manylinux2010_x86_64.whl (110.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 110.5 MB 539 bytes/s \n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from keras@ https://github.com/MarcBS/keras/archive/master.zip->nmt-keras==0.6) (3.13)\n",
            "Collecting tensorboard<1.16.0,>=1.15.0\n",
            "  Downloading tensorboard-1.15.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8 MB 55.7 MB/s \n",
            "\u001b[?25hCollecting numpy\n",
            "  Downloading numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 20.1 MB 14.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (1.12.1)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.8.1)\n",
            "Collecting tensorflow-estimator==1.15.1\n",
            "  Downloading tensorflow_estimator-1.15.1-py2.py3-none-any.whl (503 kB)\n",
            "\u001b[K     |████████████████████████████████| 503 kB 54.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (1.1.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.2.0)\n",
            "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (3.17.3)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (1.34.1)\n",
            "Collecting h5py\n",
            "  Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.9 MB 55.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.36.2)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (3.3.0)\n",
            "Collecting gast==0.2.2\n",
            "  Downloading gast-0.2.2.tar.gz (10 kB)\n",
            "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow<2->nmt-keras==0.6) (0.12.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (3.3.4)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (1.0.1)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (57.2.0)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (4.6.1)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<1.16.0,>=1.15.0->tensorflow<2->nmt-keras==0.6) (3.5.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (2.4.7)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (0.10.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->nmt-keras==0.6) (2.8.1)\n",
            "Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from multimodal-keras-wrapper->nmt-keras==0.6) (0.29.23)\n",
            "Requirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from multimodal-keras-wrapper->nmt-keras==0.6) (0.0)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.7/dist-packages (from multimodal-keras-wrapper->nmt-keras==0.6) (0.11.1)\n",
            "Collecting subword-nmt\n",
            "  Downloading subword_nmt-0.3.7-py2.py3-none-any.whl (26 kB)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->nmt-keras==0.6) (2018.9)\n",
            "Collecting portalocker==2.0.0\n",
            "  Downloading portalocker-2.0.0-py2.py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (2019.12.20)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (7.1.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (4.41.1)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->nmt-keras==0.6) (1.0.1)\n",
            "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (2.5.1)\n",
            "Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (1.1.1)\n",
            "Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (2.4.1)\n",
            "Requirement already satisfied: pillow>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->nmt-keras==0.6) (7.1.2)\n",
            "Requirement already satisfied: decorator<5,>=4.3 in /usr/local/lib/python3.7/dist-packages (from networkx>=2.0->scikit-image->nmt-keras==0.6) (4.4.2)\n",
            "Requirement already satisfied: numexpr>=2.5.2 in /usr/local/lib/python3.7/dist-packages (from tables->nmt-keras==0.6) (2.7.3)\n",
            "Building wheels for collected packages: keras, gast\n",
            "  Building wheel for keras (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for keras: filename=Keras-2.3.1.1-py3-none-any.whl size=487514 sha256=88ffe3363f1f6c6425ba4068989ef60cda757a504814c9246a8d493a2d2fbea2\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-c9zqgxqd/wheels/68/86/9b/290dd8e0919a4070424e29c34886fbcf85d437c53506723c08\n",
            "  Building wheel for gast (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gast: filename=gast-0.2.2-py3-none-any.whl size=7553 sha256=d212aa931798d55a7ccdcc7fd26d073d6dc81a375e86c5bfbb0bdc722b057587\n",
            "  Stored in directory: /root/.cache/pip/wheels/21/7f/02/420f32a803f7d0967b48dd823da3f558c5166991bfd204eef3\n",
            "Successfully built keras gast\n",
            "Installing collected packages: numpy, h5py, portalocker, keras-applications, tensorflow-estimator, tensorboard, subword-nmt, sacremoses, sacrebleu, keras, gast, tensorflow, multimodal-keras-wrapper, nmt-keras\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.19.5\n",
            "    Uninstalling numpy-1.19.5:\n",
            "      Successfully uninstalled numpy-1.19.5\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.5.0\n",
            "    Uninstalling tensorflow-estimator-2.5.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.5.0\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.5.0\n",
            "    Uninstalling tensorboard-2.5.0:\n",
            "      Successfully uninstalled tensorboard-2.5.0\n",
            "  Attempting uninstall: gast\n",
            "    Found existing installation: gast 0.4.0\n",
            "    Uninstalling gast-0.4.0:\n",
            "      Successfully uninstalled gast-0.4.0\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.5.0\n",
            "    Uninstalling tensorflow-2.5.0:\n",
            "      Successfully uninstalled tensorflow-2.5.0\n",
            "  Running setup.py develop for nmt-keras\n",
            "Successfully installed gast-0.2.2 h5py-2.10.0 keras-2.3.1.1 keras-applications-1.0.8 multimodal-keras-wrapper-3.1.6 nmt-keras-0.6 numpy-1.18.5 portalocker-2.0.0 sacrebleu-1.5.1 sacremoses-0.0.45 subword-nmt-0.3.7 tensorboard-1.15.0 tensorflow-1.15.5 tensorflow-estimator-1.15.1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "numpy"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSSh7bps4y1U"
      },
      "source": [
        "## 1. Building a Dataset model\n",
        "First, we are creating a [Dataset](https://github.com/MarcBS/multimodal_keras_wrapper/keras_wrapper/dataset.py) object (from the [Multimodal Keras Wrapper](https://github.com/MarcBS/multimodal_keras_wrapper) library). This object will be the interface between our data (text files) and the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hiY2fUFU83Rx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0d6aa942-16fa-4cef-b3e9-9ddf406ee2da"
      },
      "source": [
        "from keras_wrapper.dataset import Dataset, saveDataset\n",
        "from data_engine.prepare_data import keep_n_captions\n",
        "ds = Dataset('tutorial_dataset', 'tutorial', silence=False)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1ify05BL8_Rj"
      },
      "source": [
        "Now that we have the empty dataset, we must indicate its inputs and outputs. In our case, we'll have two different inputs and one single output:\n",
        "\n",
        "1. Outputs:\n",
        "**target_text**: Sentences in our target language.\n",
        "\n",
        "2. Inputs:\n",
        "**source_text**: Sentences in the source language.\n",
        "\n",
        "**state_below**: Sentences in the target language, but shifted one position to the right (for teacher-forcing training of the model).\n",
        "\n",
        "For setting up the outputs, we use the setOutputs function, with the appropriate parameters. Note that, when we are building the dataset for the training split, we build the vocabulary (up to 30000 words)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d1A8D0qn9IeE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "09f3fe3d-4e98-48a1-f887-ba1dbfb80f67"
      },
      "source": [
        "ds.setOutput('examples/EuTrans/training.en',\n",
        "             'train',\n",
        "             type='text',\n",
        "             id='target_text',\n",
        "             tokenization='tokenize_none',\n",
        "             build_vocabulary=True,\n",
        "             pad_on_batch=True,\n",
        "             sample_weights=True,\n",
        "             max_text_len=30,\n",
        "             max_words=30000,\n",
        "             min_occ=0)\n",
        "\n",
        "ds.setOutput('examples/EuTrans/dev.en',\n",
        "             'val',\n",
        "             type='text',\n",
        "             id='target_text',\n",
        "             pad_on_batch=True,\n",
        "             tokenization='tokenize_none',\n",
        "             sample_weights=True,\n",
        "             max_text_len=30,\n",
        "             max_words=0)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:42] \tApplying tokenization function: \"tokenize_none\".\n",
            "[30/07/2021 09:21:42] Creating vocabulary for data with data_id 'target_text'.\n",
            "[30/07/2021 09:21:42] \t Total: 513 unique words in 9900 sentences with a total of 98304 words.\n",
            "[30/07/2021 09:21:42] Creating dictionary of 30000 most common words, covering 100.0% of the text.\n",
            "[30/07/2021 09:21:42] Loaded \"train\" set outputs of data_type \"text\" with data_id \"target_text\" and length 9900.\n",
            "[30/07/2021 09:21:42] \tApplying tokenization function: \"tokenize_none\".\n",
            "[30/07/2021 09:21:42] Loaded \"val\" set outputs of data_type \"text\" with data_id \"target_text\" and length 100.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jipVyl7f9NTz"
      },
      "source": [
        "Similarly, we introduce the source text data, with the setInputs function. Again, when building the training split, we must construct the vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f72G9mHX9PPq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a45b9c16-f694-494e-85f2-4177ba84a5d5"
      },
      "source": [
        "ds.setInput('examples/EuTrans/training.es',\n",
        "            'train',\n",
        "            type='text',\n",
        "            id='source_text',\n",
        "            pad_on_batch=True,\n",
        "            tokenization='tokenize_none',\n",
        "            build_vocabulary=True,\n",
        "            fill='end',\n",
        "            max_text_len=30,\n",
        "            max_words=30000,\n",
        "            min_occ=0)\n",
        "ds.setInput('examples/EuTrans/dev.es',\n",
        "            'val',\n",
        "            type='text',\n",
        "            id='source_text',\n",
        "            pad_on_batch=True,\n",
        "            tokenization='tokenize_none',\n",
        "            fill='end',\n",
        "            max_text_len=30,\n",
        "            min_occ=0)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:42] \tApplying tokenization function: \"tokenize_none\".\n",
            "[30/07/2021 09:21:43] Creating vocabulary for data with data_id 'source_text'.\n",
            "[30/07/2021 09:21:43] \t Total: 686 unique words in 9900 sentences with a total of 96172 words.\n",
            "[30/07/2021 09:21:43] Creating dictionary of 30000 most common words, covering 100.0% of the text.\n",
            "[30/07/2021 09:21:43] Loaded \"train\" set inputs of data_type \"text\" with data_id \"source_text\" and length 9900.\n",
            "[30/07/2021 09:21:43] \tApplying tokenization function: \"tokenize_none\".\n",
            "[30/07/2021 09:21:43] Loaded \"val\" set inputs of data_type \"text\" with data_id \"source_text\" and length 100.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHvZimmm9U0p"
      },
      "source": [
        "...and for the 'state_below' data. Note that: 1) The offset flat is set to 1, which means that the text will be shifted to the right 1 position. 2) During sampling time, we won't have this input. Hence, we 'hack' the dataset model by inserting an artificial input, of type 'ghost' for the validation split."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YiTM3y449ZFL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c3edda7f-2a14-4bd9-a000-40d01e5ba128"
      },
      "source": [
        "ds.setInput('examples/EuTrans/training.en',\n",
        "            'train',\n",
        "            type='text',\n",
        "            id='state_below',\n",
        "            required=False,\n",
        "            tokenization='tokenize_none',\n",
        "            pad_on_batch=True,\n",
        "            build_vocabulary='target_text',\n",
        "            offset=1,\n",
        "            fill='end',\n",
        "            max_text_len=30,\n",
        "            max_words=30000)\n",
        "ds.setInput(None,\n",
        "            'val',\n",
        "            type='ghost',\n",
        "            id='state_below',\n",
        "            required=False)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:43] \tApplying tokenization function: \"tokenize_none\".\n",
            "[30/07/2021 09:21:43] \tReusing vocabulary named \"target_text\" for data with data_id \"state_below\".\n",
            "[30/07/2021 09:21:43] Loaded \"train\" set inputs of data_type \"text\" with data_id \"state_below\" and length 9900.\n",
            "[30/07/2021 09:21:43] Loaded \"val\" set inputs of data_type \"ghost\" with data_id \"state_below\" and length 100.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9H-1qb1ksD5Y"
      },
      "source": [
        "We can also keep the literal source words (for replacing unknown words)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AkC4AUjGsLYd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1b979608-39e5-4f6e-d4f8-1709d7bd3487"
      },
      "source": [
        "  for split, input_text_filename in zip(['train', 'val'], ['examples/EuTrans/training.es', 'examples/EuTrans/dev.es']):\n",
        "    ds.setRawInput(input_text_filename,\n",
        "                  split,\n",
        "                  type='file-name',\n",
        "                  id='raw_source_text',\n",
        "                  overwrite_split=True)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:43] Loaded \"train\" set inputs of type \"file-name\" with id \"raw_source_text\".\n",
            "[30/07/2021 09:21:43] Loaded \"val\" set inputs of type \"file-name\" with id \"raw_source_text\".\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lcYiuysd9cK4"
      },
      "source": [
        "We also need to match the references with the inputs. Since we only have one reference per input sample, we set `repeat=1`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s94NfwBl9sZP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a0d26773-e02f-4f4e-f33c-41703bd36dd4"
      },
      "source": [
        "keep_n_captions(ds, repeat=1, n=1, set_names=['val'])\n"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:43] Keeping 1 captions per input on the val set.\n",
            "[30/07/2021 09:21:43] Samples reduced to 100 in val set.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sDYhlBpR9vgi"
      },
      "source": [
        "Finally, we can save our dataset instance for using in other experiments:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sS5vZ5zp9tkG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd4d7d5d-5c0e-4c52-d475-698fe1aecdf4"
      },
      "source": [
        "saveDataset(ds, 'datasets')\n"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:43] <<< creating directory datasets ... >>>\n",
            "[30/07/2021 09:21:43] <<< Saving Dataset instance to datasets/Dataset_tutorial_dataset.pkl ... >>>\n",
            "[30/07/2021 09:21:43] <<< Dataset instance saved >>>\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yj8aYICU-eLH"
      },
      "source": [
        "## 2. Creating and training a Neural Translation Model\n",
        "Now, we'll create and train a Neural Machine Translation (NMT) model. Since there is a significant number of hyperparameters, we'll use the default ones, specified in the `config.py` file. Note that almost every hardcoded parameter is automatically set from config if we run  `main.py `.\n",
        "\n",
        "We'll create an `'AttentionRNNEncoderDecoder'` (a LSTM encoder-decoder with attention mechanism). Refer to the [`model_zoo.py`](https://github.com/lvapeab/nmt-keras/blob/master/nmt_keras/model_zoo.py) file for other models (e.g. Transformer). \n",
        "\n",
        "So first, let's import the model and the hyperparameters. We'll also load the dataset we stored in the previous section (not necessary as it is in memory, but as a demonstration):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TszghyVO_M0B",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e25c7d2-063d-4c30-9234-5403535928b5"
      },
      "source": [
        "from config import load_parameters\n",
        "from nmt_keras.model_zoo import TranslationModel\n",
        "from keras_wrapper.cnn_model import loadModel\n",
        "from keras_wrapper.dataset import loadDataset\n",
        "from keras_wrapper.extra.callbacks import PrintPerformanceMetricOnEpochEndOrEachNUpdates\n",
        "params = load_parameters()\n",
        "dataset = loadDataset('datasets/Dataset_tutorial_dataset.pkl')\n"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:45] <<< Loading Dataset instance from datasets/Dataset_tutorial_dataset.pkl ... >>>\n",
            "[30/07/2021 09:21:45] <<< Dataset instance loaded >>>\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7MHVvMDYFmcQ"
      },
      "source": [
        "Since the number of words in the dataset may be unknown beforehand, we must update the params information according to the dataset instance:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h7FIJGybFm7C"
      },
      "source": [
        "params['INPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len['source_text']\n",
        "params['OUTPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len['target_text']"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c3a9XanxFpp7"
      },
      "source": [
        "Now, we create a `TranslationModel` instance:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2RYqrU6VFr8U",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b2a02b5c-6c76-40c8-bd7c-f0a134aac29f"
      },
      "source": [
        "params['MODEL_TYPE'] = 'AttentionRNNEncoderDecoder' #  Supported models: 'AttentionRNNEncoderDecoder' and 'Transformer'.\n",
        "nmt_model = TranslationModel(params,\n",
        "                             model_type=params['MODEL_TYPE'], \n",
        "                             model_name='tutorial_model',\n",
        "                             vocabularies=dataset.vocabulary,\n",
        "                             store_path='trained_models/tutorial_model/',\n",
        "                             verbose=True)\n"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:45] <<< Building AttentionRNNEncoderDecoder Translation_Model >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:650: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:45] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:650: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:4786: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:45] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:4786: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:157: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:45] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:157: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:3561: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:45] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:3561: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "-----------------------------------------------------------------------------------\n",
            "\t\tTranslationModel instance\n",
            "-----------------------------------------------------------------------------------\n",
            "_model_type: AttentionRNNEncoderDecoder\n",
            "name: tutorial_model\n",
            "model_path: trained_models/tutorial_model/\n",
            "verbose: True\n",
            "\n",
            "Params:\n",
            "\tACCUMULATE_GRADIENTS: 1\n",
            "\tADDITIONAL_OUTPUT_MERGE_MODE: Add\n",
            "\tALIGN_FROM_RAW: True\n",
            "\tALPHA_FACTOR: 0.6\n",
            "\tAMSGRAD: False\n",
            "\tAPPLY_DETOKENIZATION: False\n",
            "\tATTENTION_DROPOUT_P: 0.0\n",
            "\tATTENTION_MODE: add\n",
            "\tATTENTION_SIZE: 32\n",
            "\tBATCH_NORMALIZATION_MODE: 1\n",
            "\tBATCH_SIZE: 50\n",
            "\tBEAM_SEARCH: True\n",
            "\tBEAM_SIZE: 6\n",
            "\tBETA_1: 0.9\n",
            "\tBETA_2: 0.999\n",
            "\tBIDIRECTIONAL_DEEP_ENCODER: True\n",
            "\tBIDIRECTIONAL_ENCODER: True\n",
            "\tBIDIRECTIONAL_MERGE_MODE: concat\n",
            "\tBPE_CODES_PATH: examples/EuTrans//training_codes.joint\n",
            "\tCLASSIFIER_ACTIVATION: softmax\n",
            "\tCLIP_C: 5.0\n",
            "\tCLIP_V: 0.0\n",
            "\tCOVERAGE_NORM_FACTOR: 0.2\n",
            "\tCOVERAGE_PENALTY: False\n",
            "\tDATASET_NAME: EuTrans\n",
            "\tDATASET_STORE_PATH: datasets/\n",
            "\tDATA_AUGMENTATION: False\n",
            "\tDATA_ROOT_PATH: examples/EuTrans/\n",
            "\tDECODER_HIDDEN_SIZE: 32\n",
            "\tDECODER_RNN_TYPE: ConditionalLSTM\n",
            "\tDEEP_OUTPUT_LAYERS: [('linear', 32)]\n",
            "\tDETOKENIZATION_METHOD: detokenize_none\n",
            "\tDOUBLE_STOCHASTIC_ATTENTION_REG: 0.0\n",
            "\tDROPOUT_P: 0.0\n",
            "\tEARLY_STOP: True\n",
            "\tEMBEDDINGS_FREQ: 1\n",
            "\tENCODER_HIDDEN_SIZE: 32\n",
            "\tENCODER_RNN_TYPE: LSTM\n",
            "\tEPOCHS_FOR_SAVE: 1\n",
            "\tEPSILON: 1e-08\n",
            "\tEVAL_EACH: 1\n",
            "\tEVAL_EACH_EPOCHS: True\n",
            "\tEVAL_ON_SETS: ['val']\n",
            "\tEXTRA_NAME: \n",
            "\tFF_SIZE: 128\n",
            "\tFILL: end\n",
            "\tFORCE_RELOAD_VOCABULARY: False\n",
            "\tGLOSSARY: None\n",
            "\tGRU_RESET_AFTER: True\n",
            "\tHEURISTIC: 0\n",
            "\tHOMOGENEOUS_BATCHES: False\n",
            "\tINIT_ATT: glorot_uniform\n",
            "\tINIT_FUNCTION: glorot_uniform\n",
            "\tINIT_LAYERS: ['tanh']\n",
            "\tINNER_INIT: orthogonal\n",
            "\tINPUTS_IDS_DATASET: ['source_text', 'state_below']\n",
            "\tINPUTS_IDS_MODEL: ['source_text', 'state_below']\n",
            "\tINPUTS_TYPES_DATASET: ['text-features', 'text-features']\n",
            "\tINPUT_VOCABULARY_SIZE: 689\n",
            "\tJOINT_BATCHES: 4\n",
            "\tKERAS_METRICS: ['perplexity']\n",
            "\tLABEL_SMOOTHING: 0.0\n",
            "\tLENGTH_NORM_FACTOR: 0.2\n",
            "\tLENGTH_PENALTY: False\n",
            "\tLOG_DIR: tensorboard_logs\n",
            "\tLOSS: categorical_crossentropy\n",
            "\tLR: 0.001\n",
            "\tLR_DECAY: None\n",
            "\tLR_GAMMA: 0.8\n",
            "\tLR_HALF_LIFE: 100\n",
            "\tLR_REDUCER_EXP_BASE: -0.5\n",
            "\tLR_REDUCER_TYPE: exponential\n",
            "\tLR_REDUCE_EACH_EPOCHS: False\n",
            "\tLR_START_REDUCTION_ON_EPOCH: 0\n",
            "\tMAPPING: examples/EuTrans//mapping.es_en.pkl\n",
            "\tMAXLEN_GIVEN_X: True\n",
            "\tMAXLEN_GIVEN_X_FACTOR: 2\n",
            "\tMAX_EPOCH: 500\n",
            "\tMAX_INPUT_TEXT_LEN: 50\n",
            "\tMAX_OUTPUT_TEXT_LEN: 50\n",
            "\tMAX_OUTPUT_TEXT_LEN_TEST: 150\n",
            "\tMAX_PLOT_Y: 100.0\n",
            "\tMETRICS: ['sacrebleu', 'perplexity']\n",
            "\tMINLEN_GIVEN_X: True\n",
            "\tMINLEN_GIVEN_X_FACTOR: 3\n",
            "\tMIN_DELTA: 0.0\n",
            "\tMIN_LR: 1e-09\n",
            "\tMIN_OCCURRENCES_INPUT_VOCAB: 0\n",
            "\tMIN_OCCURRENCES_OUTPUT_VOCAB: 0\n",
            "\tMODE: training\n",
            "\tMODEL_NAME: EuTrans_esen_AttentionRNNEncoderDecoder_src_emb_32_bidir_True_enc_LSTM_32_dec_ConditionalLSTM_32_deepout_linear_trg_emb_32_Adam_0.001\n",
            "\tMODEL_SIZE: 32\n",
            "\tMODEL_TYPE: AttentionRNNEncoderDecoder\n",
            "\tMOMENTUM: 0.0\n",
            "\tMULTIHEAD_ATTENTION_ACTIVATION: linear\n",
            "\tNESTEROV_MOMENTUM: False\n",
            "\tNOISE_AMOUNT: 0.01\n",
            "\tNORMALIZE_SAMPLING: False\n",
            "\tN_GPUS: 1\n",
            "\tN_HEADS: 8\n",
            "\tN_LAYERS_DECODER: 1\n",
            "\tN_LAYERS_ENCODER: 1\n",
            "\tN_SAMPLES: 5\n",
            "\tOPTIMIZED_SEARCH: True\n",
            "\tOPTIMIZER: Adam\n",
            "\tOUTPUTS_IDS_DATASET: ['target_text']\n",
            "\tOUTPUTS_IDS_MODEL: ['target_text']\n",
            "\tOUTPUTS_TYPES_DATASET: ['text-features']\n",
            "\tOUTPUT_VOCABULARY_SIZE: 516\n",
            "\tPAD_ON_BATCH: True\n",
            "\tPARALLEL_LOADERS: 1\n",
            "\tPATIENCE: 10\n",
            "\tPLOT_EVALUATION: False\n",
            "\tPOS_UNK: True\n",
            "\tREBUILD_DATASET: True\n",
            "\tRECURRENT_DROPOUT_P: 0.0\n",
            "\tRECURRENT_INPUT_DROPOUT_P: 0.0\n",
            "\tRECURRENT_WEIGHT_DECAY: 0.0\n",
            "\tREGULARIZATION_FN: L2\n",
            "\tRELOAD: 0\n",
            "\tRELOAD_EPOCH: True\n",
            "\tRHO: 0.9\n",
            "\tSAMPLE_EACH_UPDATES: 300\n",
            "\tSAMPLE_ON_SETS: ['train', 'val']\n",
            "\tSAMPLE_WEIGHTS: True\n",
            "\tSAMPLING: max_likelihood\n",
            "\tSAMPLING_SAVE_MODE: list\n",
            "\tSAVE_EACH_EVALUATION: True\n",
            "\tSCALE_SOURCE_WORD_EMBEDDINGS: False\n",
            "\tSCALE_TARGET_WORD_EMBEDDINGS: False\n",
            "\tSEARCH_PRUNING: False\n",
            "\tSKIP_VECTORS_HIDDEN_SIZE: 32\n",
            "\tSKIP_VECTORS_SHARED_ACTIVATION: tanh\n",
            "\tSOURCE_TEXT_EMBEDDING_SIZE: 32\n",
            "\tSRC_LAN: es\n",
            "\tSRC_PRETRAINED_VECTORS: None\n",
            "\tSRC_PRETRAINED_VECTORS_TRAINABLE: True\n",
            "\tSTART_EVAL_ON_EPOCH: 1\n",
            "\tSTART_SAMPLING_ON_EPOCH: 1\n",
            "\tSTOP_METRIC: Bleu_4\n",
            "\tSTORE_PATH: trained_models/EuTrans_esen_AttentionRNNEncoderDecoder_src_emb_32_bidir_True_enc_LSTM_32_dec_ConditionalLSTM_32_deepout_linear_trg_emb_32_Adam_0.001/\n",
            "\tTARGET_TEXT_EMBEDDING_SIZE: 32\n",
            "\tTASK_NAME: EuTrans\n",
            "\tTEMPERATURE: 1\n",
            "\tTENSORBOARD: True\n",
            "\tTEXT_FILES: {'train': 'training.', 'val': 'dev.', 'test': 'test.'}\n",
            "\tTIE_EMBEDDINGS: False\n",
            "\tTOKENIZATION_METHOD: tokenize_none\n",
            "\tTOKENIZE_HYPOTHESES: True\n",
            "\tTOKENIZE_REFERENCES: True\n",
            "\tTRAINABLE_DECODER: True\n",
            "\tTRAINABLE_ENCODER: True\n",
            "\tTRAIN_ON_TRAINVAL: False\n",
            "\tTRG_LAN: en\n",
            "\tTRG_PRETRAINED_VECTORS: None\n",
            "\tTRG_PRETRAINED_VECTORS_TRAINABLE: True\n",
            "\tUSE_BATCH_NORMALIZATION: True\n",
            "\tUSE_CUDNN: False\n",
            "\tUSE_L1: False\n",
            "\tUSE_L2: False\n",
            "\tUSE_NOISE: False\n",
            "\tUSE_PRELU: False\n",
            "\tUSE_TF_OPTIMIZER: True\n",
            "\tVERBOSE: 1\n",
            "\tWARMUP_EXP: -1.5\n",
            "\tWEIGHT_DECAY: 0.0001\n",
            "\tWRITE_VALID_SAMPLES: True\n",
            "-----------------------------------------------------------------------------------\n",
            "Model: \"tutorial_model_training\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "source_text (InputLayer)        (None, None)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "source_word_embedding (Embeddin (None, None, 32)     22048       source_text[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "src_embedding_batch_normalizati (None, None, 32)     128         source_word_embedding[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "remove_mask_1 (RemoveMask)      (None, None, 32)     0           src_embedding_batch_normalization\n",
            "__________________________________________________________________________________________________\n",
            "bidirectional_encoder_LSTM (Bid (None, None, 64)     16640       remove_mask_1[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "annotations_batch_normalization (None, None, 64)     256         bidirectional_encoder_LSTM[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "source_text_mask (GetMask)      (None, None, 32)     0           src_embedding_batch_normalization\n",
            "__________________________________________________________________________________________________\n",
            "annotations (ApplyMask)         (None, None, 64)     0           annotations_batch_normalization[0\n",
            "                                                                 source_text_mask[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "state_below (InputLayer)        (None, None)         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "ctx_mean (MaskedMean)           (None, 64)           0           annotations[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "target_word_embedding (Embeddin (None, None, 32)     16512       state_below[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "initial_state (Dense)           (None, 32)           2080        ctx_mean[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "initial_memory (Dense)          (None, 32)           2080        ctx_mean[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "state_below_batch_normalization (None, None, 32)     128         target_word_embedding[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "initial_state_batch_normalizati (None, 32)           128         initial_state[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "initial_memory_batch_normalizat (None, 32)           128         initial_memory[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "decoder_AttConditionalLSTMCond  [(None, None, 32), ( 23873       state_below_batch_normalization[0\n",
            "                                                                 annotations[0][0]                \n",
            "                                                                 initial_state_batch_normalization\n",
            "                                                                 initial_memory_batch_normalizatio\n",
            "__________________________________________________________________________________________________\n",
            "proj_h0_batch_normalization (Ba (None, None, 32)     128         decoder_AttConditionalLSTMCond[0]\n",
            "__________________________________________________________________________________________________\n",
            "logit_ctx (TimeDistributed)     (None, None, 32)     2080        decoder_AttConditionalLSTMCond[0]\n",
            "__________________________________________________________________________________________________\n",
            "logit_lstm (TimeDistributed)    (None, None, 32)     1056        proj_h0_batch_normalization[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "permute_general_1 (PermuteGener (None, None, 32)     0           logit_ctx[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "logit_emb (TimeDistributed)     (None, None, 32)     1056        state_below_batch_normalization[0\n",
            "__________________________________________________________________________________________________\n",
            "out_layer_mlp_batch_normalizati (None, None, 32)     128         logit_lstm[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "out_layer_ctx_batch_normalizati (None, None, 32)     128         permute_general_1[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "out_layer_emb_batch_normalizati (None, None, 32)     128         logit_emb[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "additional_input (Add)          (None, None, 32)     0           out_layer_mlp_batch_normalization\n",
            "                                                                 out_layer_ctx_batch_normalization\n",
            "                                                                 out_layer_emb_batch_normalization\n",
            "__________________________________________________________________________________________________\n",
            "activation_1 (Activation)       (None, None, 32)     0           additional_input[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "linear_0 (TimeDistributed)      (None, None, 32)     1056        activation_1[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "out_layer_linear_0_batch_normal (None, None, 32)     128         linear_0[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "target_text (TimeDistributed)   (None, None, 516)    17028       out_layer_linear_0_batch_normaliz\n",
            "==================================================================================================\n",
            "Total params: 106,917\n",
            "Trainable params: 106,213\n",
            "Non-trainable params: 704\n",
            "__________________________________________________________________________________________________\n",
            "WARNING:tensorflow:From /content/nmt-keras/nmt_keras/model_zoo.py:213: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:46] From /content/nmt-keras/nmt_keras/model_zoo.py:213: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "[30/07/2021 09:21:46] Preparing optimizer and compiling. Optimizer configuration: \n",
            "\t LR: 0.001\n",
            "\t LOSS: categorical_crossentropy\n",
            "\t BETA_1: 0.9\n",
            "\t BETA_2: 0.999\n",
            "\t EPSILON: 1e-08\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:1192: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:46] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:1192: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pKQqWOneGD_3"
      },
      "source": [
        "Next, we must define the inputs and outputs mapping from our Dataset instance to our model:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VqEqWxYHGIM5"
      },
      "source": [
        "\n",
        "inputMapping = dict()\n",
        "for i, id_in in enumerate(params['INPUTS_IDS_DATASET']):\n",
        "    pos_source = dataset.ids_inputs.index(id_in)\n",
        "    id_dest = nmt_model.ids_inputs[i]\n",
        "    inputMapping[id_dest] = pos_source\n",
        "nmt_model.setInputsMapping(inputMapping)\n",
        "\n",
        "outputMapping = dict()\n",
        "for i, id_out in enumerate(params['OUTPUTS_IDS_DATASET']):\n",
        "    pos_target = dataset.ids_outputs.index(id_out)\n",
        "    id_dest = nmt_model.ids_outputs[i]\n",
        "    outputMapping[id_dest] = pos_target\n",
        "nmt_model.setOutputsMapping(outputMapping)\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKPW2hv8GKMj"
      },
      "source": [
        "We can add some callbacks for controlling the training (e.g. Sampling each N updates, early stop, learning rate annealing...). For instance, let's build a sampling callback. After each epoch, it will compute the BLEU scores on the development set using the sacreBLEU package. We need to pass some configuration variables to the callback (in the extra_vars dictionary):\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7MtMvSoAGNHb"
      },
      "source": [
        "is_transformer = params.get('ATTEND_ON_OUTPUT', 'transformer' in params['MODEL_TYPE'].lower())\n",
        "search_params = {\n",
        "    'language': 'en',\n",
        "    'tokenize_f': eval('dataset.' + 'tokenize_none'),\n",
        "    'beam_size': 12,\n",
        "    'optimized_search': True,\n",
        "    'model_inputs': params['INPUTS_IDS_MODEL'],\n",
        "    'model_outputs': params['OUTPUTS_IDS_MODEL'],\n",
        "    'dataset_inputs':  params['INPUTS_IDS_DATASET'],\n",
        "    'dataset_outputs':  params['OUTPUTS_IDS_DATASET'],\n",
        "    'n_parallel_loaders': 1,\n",
        "    'maxlen': 50,\n",
        "    'normalize_probs': True,\n",
        "    'pos_unk': True and not is_transformer,  # Pos_unk is unimplemented for transformer models\n",
        "    'heuristic': 0,\n",
        "    'state_below_maxlen': -1,\n",
        "    'attend_on_output': is_transformer,\n",
        "    'val': {'references': dataset.extra_variables['val']['target_text']}\n",
        "  }\n",
        "\n",
        "vocab = dataset.vocabulary['target_text']['idx2words']\n",
        "callbacks = []\n",
        "input_text_id = params['INPUTS_IDS_DATASET'][0]\n",
        "\n",
        "callbacks.append(PrintPerformanceMetricOnEpochEndOrEachNUpdates(nmt_model,\n",
        "                                                                dataset,\n",
        "                                                                gt_id='target_text',\n",
        "                                                                metric_name=['sacrebleu'],\n",
        "                                                                set_name=['val'],\n",
        "                                                                batch_size=50,\n",
        "                                                                each_n_epochs=1,\n",
        "                                                                extra_vars=search_params,\n",
        "                                                                reload_epoch=0,\n",
        "                                                                is_text=True,\n",
        "                                                                input_text_id=input_text_id,\n",
        "                                                                index2word_y=vocab,\n",
        "                                                                sampling_type='max_likelihood',\n",
        "                                                                beam_search=True,\n",
        "                                                                save_path=nmt_model.model_path,\n",
        "                                                                start_eval_on_epoch=0,\n",
        "                                                                write_samples=True,\n",
        "                                                                write_type='list',\n",
        "                                                                verbose=True))"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vo-kOSLlGQux"
      },
      "source": [
        "Now we are ready to train. Let's set up some training parameters...\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_-oborMLGUMP"
      },
      "source": [
        "training_params = {'n_epochs': 4,\n",
        "                   'batch_size': 50,\n",
        "                   'maxlen': 30,\n",
        "                   'epochs_for_save': 1,\n",
        "                   'verbose': 1,\n",
        "                   'eval_on_sets': [], \n",
        "                   'n_parallel_loaders': 1,\n",
        "                   'extra_callbacks': callbacks,\n",
        "                   'reload_epoch': 0,\n",
        "                   'epoch_offset': 0}\n"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z7m3oR5RGVDp"
      },
      "source": [
        "And train!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PcyAKL4cGai4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "786dd9c0-7468-4965-b6e1-179611602a59"
      },
      "source": [
        "nmt_model.trainNet(dataset, training_params)\n"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:46] <<< Training model >>>\n",
            "[30/07/2021 09:21:46] Training parameters: { \n",
            "\tbatch_size: 50\n",
            "\tclass_weights: None\n",
            "\tda_enhance_list: []\n",
            "\tda_patch_type: resize_and_rndcrop\n",
            "\tdata_augmentation: False\n",
            "\teach_n_epochs: 1\n",
            "\tepoch_offset: 0\n",
            "\tepochs_for_save: 1\n",
            "\teval_on_epochs: True\n",
            "\teval_on_sets: []\n",
            "\textra_callbacks: [<keras_wrapper.extra.callbacks.EvalPerformance object at 0x7fda025085d0>]\n",
            "\thomogeneous_batches: False\n",
            "\tinitial_lr: 1.0\n",
            "\tjoint_batches: 4\n",
            "\tlr_decay: None\n",
            "\tlr_gamma: 0.1\n",
            "\tlr_half_life: 50000\n",
            "\tlr_reducer_exp_base: 0.5\n",
            "\tlr_reducer_type: linear\n",
            "\tlr_warmup_exp: -1.5\n",
            "\tmaxlen: 30\n",
            "\tmean_substraction: False\n",
            "\tmetric_check: None\n",
            "\tmin_delta: 0.0\n",
            "\tmin_lr: 1e-09\n",
            "\tn_epochs: 4\n",
            "\tn_gpus: 1\n",
            "\tn_parallel_loaders: 1\n",
            "\tnormalization_type: None\n",
            "\tnormalize: False\n",
            "\tnum_iterations_val: None\n",
            "\tpatience: 0\n",
            "\tpatience_check_split: val\n",
            "\treduce_each_epochs: True\n",
            "\treload_epoch: 0\n",
            "\tshuffle: True\n",
            "\tstart_eval_on_epoch: 0\n",
            "\tstart_reduction_on_epoch: 0\n",
            "\ttensorboard: False\n",
            "\ttensorboard_params: {'log_dir': 'tensorboard_logs', 'histogram_freq': 0, 'batch_size': 50, 'write_graph': True, 'write_grads': False, 'write_images': False, 'embeddings_freq': 0, 'embeddings_layer_names': None, 'embeddings_metadata': None, 'update_freq': 'epoch'}\n",
            "\tverbose: 1\n",
            "\two_da_patch_type: whole\n",
            "}\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:3315: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:50] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:3315: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:292: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:50] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:292: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:299: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:50] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:299: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:312: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:50] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:312: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:321: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:50] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:321: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:328: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:21:51] From /usr/local/lib/python3.7/dist-packages/keras/backend/tensorflow_backend.py:328: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/4\n",
            "198/198 [==============================] - 25s 127ms/step - loss: 1.8775 - perplexity: 124.4208\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:22:16] <<< Saving model to trained_models/tutorial_model/epoch_1 ... >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/keras/engine/saving.py:165: UserWarning: TensorFlow optimizers do not make it possible to access optimizer attributes or optimizer state after instantiation. As a result, we cannot save the optimizer as part of the model save file.You will have to compile your model again after loading it. Prefer using a Keras optimizer instead (see keras.io/optimizers).\n",
            "  'TensorFlow optimizers do not '\n",
            "[30/07/2021 09:22:19] <<< Model saved >>>\n",
            "\n",
            "[30/07/2021 09:22:19] <<< Predicting outputs of val set >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            " Total cost: 483.882177 \t Average cost: 4.838822\n",
            "The sampling took: 9.416357 secs (Speed: 0.094164 sec/sample)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:22:28] Prediction output 0: target_text (text)\n",
            "[30/07/2021 09:22:28] Decoding beam search prediction ...\n",
            "[30/07/2021 09:22:28] Using heuristic 0\n",
            "[30/07/2021 09:22:28] Evaluating on metric sacrebleu\n",
            "[30/07/2021 09:22:28] Computing SacreBleu scores on the val split...\n",
            "[30/07/2021 09:22:28] Bleu_4: 0.17795412942601663\n",
            "[30/07/2021 09:22:28] Done evaluating on metric sacrebleu\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:22:28] \n",
            "<<< Progress plot saved in trained_models/tutorial_model/epoch_1.jpg >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 2/4\n",
            "198/198 [==============================] - 23s 114ms/step - loss: 0.6594 - perplexity: 25.6357\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:22:51] <<< Saving model to trained_models/tutorial_model/epoch_2 ... >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:22:51] <<< Model saved >>>\n",
            "\n",
            "[30/07/2021 09:22:51] <<< Predicting outputs of val set >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            " Total cost: 411.832776 \t Average cost: 4.118328\n",
            "The sampling took: 6.167054 secs (Speed: 0.061671 sec/sample)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:22:57] Prediction output 0: target_text (text)\n",
            "[30/07/2021 09:22:57] Decoding beam search prediction ...\n",
            "[30/07/2021 09:22:57] Using heuristic 0\n",
            "[30/07/2021 09:22:57] Evaluating on metric sacrebleu\n",
            "[30/07/2021 09:22:57] Computing SacreBleu scores on the val split...\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:22:57] Bleu_4: 30.492151506919626\n",
            "[30/07/2021 09:22:57] Done evaluating on metric sacrebleu\n",
            "[30/07/2021 09:22:57] \n",
            "<<< Progress plot saved in trained_models/tutorial_model/epoch_2.jpg >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 3/4\n",
            "198/198 [==============================] - 23s 117ms/step - loss: 0.3967 - perplexity: 15.4490\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:21] <<< Saving model to trained_models/tutorial_model/epoch_3 ... >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:21] <<< Model saved >>>\n",
            "\n",
            "[30/07/2021 09:23:21] <<< Predicting outputs of val set >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            " Total cost: 313.784344 \t Average cost: 3.137843\n",
            "The sampling took: 5.905695 secs (Speed: 0.059057 sec/sample)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:27] Prediction output 0: target_text (text)\n",
            "[30/07/2021 09:23:27] Decoding beam search prediction ...\n",
            "[30/07/2021 09:23:27] Using heuristic 0\n",
            "[30/07/2021 09:23:27] Evaluating on metric sacrebleu\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:27] Computing SacreBleu scores on the val split...\n",
            "[30/07/2021 09:23:27] Bleu_4: 63.976769387761614\n",
            "[30/07/2021 09:23:27] Done evaluating on metric sacrebleu\n",
            "[30/07/2021 09:23:27] \n",
            "<<< Progress plot saved in trained_models/tutorial_model/epoch_3.jpg >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Epoch 4/4\n",
            "198/198 [==============================] - 23s 119ms/step - loss: 0.2952 - perplexity: 9.0873\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:50] <<< Saving model to trained_models/tutorial_model/epoch_4 ... >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:50] <<< Model saved >>>\n",
            "\n",
            "[30/07/2021 09:23:50] <<< Predicting outputs of val set >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            " Total cost: 247.694837 \t Average cost: 2.476948\n",
            "The sampling took: 5.517235 secs (Speed: 0.055172 sec/sample)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:56] Prediction output 0: target_text (text)\n",
            "[30/07/2021 09:23:56] Decoding beam search prediction ...\n",
            "[30/07/2021 09:23:56] Using heuristic 0\n",
            "[30/07/2021 09:23:56] Evaluating on metric sacrebleu\n",
            "[30/07/2021 09:23:56] Computing SacreBleu scores on the val split...\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:56] Bleu_4: 75.2230275944924\n",
            "[30/07/2021 09:23:56] Done evaluating on metric sacrebleu\n",
            "[30/07/2021 09:23:56] \n",
            "<<< Progress plot saved in trained_models/tutorial_model/epoch_4.jpg >>>\n",
            "[30/07/2021 09:23:56] <<< Finished training model >>>\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hq3_MyX3A4XV"
      },
      "source": [
        "## 3. Decoding with a trained Neural Machine Translation Model\n",
        "\n",
        "Now, we'll load from disk the model we just trained and we'll apply it for translating new text. In this case, we want to translate the 'test' split from our dataset.\n",
        "\n",
        "Since we want to translate a new data split ('test') we must add it to the dataset instance, just as we did before (at the first tutorial). In case we also had the refences of the test split and we wanted to evaluate it, we can add it to the dataset. Note that this is not mandatory and we could just predict without evaluating."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2H-jXRq4BGm_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b0c7b089-ec6f-4519-d6c5-eba3caaf6784"
      },
      "source": [
        "dataset.setInput('examples/EuTrans/test.es',\n",
        "            'test',\n",
        "            type='text',\n",
        "            id='source_text',\n",
        "            pad_on_batch=True,\n",
        "            tokenization='tokenize_none',\n",
        "            fill='end',\n",
        "            max_text_len=30,\n",
        "            min_occ=0)\n",
        "\n",
        "dataset.setInput(None,\n",
        "            'test',\n",
        "            type='ghost',\n",
        "            id='state_below',\n",
        "            required=False)\n",
        "\n",
        "dataset.setRawInput('examples/EuTrans/test.es',\n",
        "              'test',\n",
        "              type='file-name',\n",
        "              id='raw_source_text',\n",
        "              overwrite_split=True)\n"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:56] \tApplying tokenization function: \"tokenize_none\".\n",
            "[30/07/2021 09:23:56] Loaded \"test\" set inputs of data_type \"text\" with data_id \"source_text\" and length 2996.\n",
            "[30/07/2021 09:23:56] Loaded \"test\" set inputs of data_type \"ghost\" with data_id \"state_below\" and length 2996.\n",
            "[30/07/2021 09:23:56] Loaded \"test\" set inputs of type \"file-name\" with id \"raw_source_text\".\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wUZveIgLCzlq"
      },
      "source": [
        "Now, let's load the translation model. Suppose we want to load the model saved at the end of the epoch 4:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8hgOSknZC2lh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "97d1c533-fb9b-45c4-a92f-89a1be4829fe"
      },
      "source": [
        "params['INPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len[params['INPUTS_IDS_DATASET'][0]]\n",
        "params['OUTPUT_VOCABULARY_SIZE'] = dataset.vocabulary_len[params['OUTPUTS_IDS_DATASET'][0]]\n",
        "\n",
        "# Load model\n",
        "nmt_model = loadModel('trained_models/tutorial_model', 4)\n"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:23:56] <<< Loading model from trained_models/tutorial_model/epoch_4_Model_Wrapper.pkl ... >>>\n",
            "[30/07/2021 09:23:56] <<< Loading model from trained_models/tutorial_model/epoch_4.h5 ... >>>\n",
            "[30/07/2021 09:23:58] <<< Loading optimized model... >>>\n",
            "[30/07/2021 09:24:03] <<< Optimized model loaded. >>>\n",
            "[30/07/2021 09:24:03] <<< Model loaded in 6.4403 seconds. >>>\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "piDc_y0pC5la"
      },
      "source": [
        "Once we loaded the model, we just have to invoke the sampling method (in this case, the Beam Search algorithm) for the 'test' split:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2FBT1HWYC9ip",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "170a2ad6-f316-474c-bc9f-b10e61235522"
      },
      "source": [
        "is_transformer = params.get('ATTEND_ON_OUTPUT', 'transformer' in params['MODEL_TYPE'].lower())\n",
        "\n",
        "params_prediction = {\n",
        "    'language': 'en',\n",
        "    'tokenize_f': eval('dataset.' + 'tokenize_none'),\n",
        "    'beam_size': 12,\n",
        "    'optimized_search': True,\n",
        "    'model_inputs': params['INPUTS_IDS_MODEL'],\n",
        "    'model_outputs': params['OUTPUTS_IDS_MODEL'],\n",
        "    'dataset_inputs':  params['INPUTS_IDS_DATASET'],\n",
        "    'dataset_outputs':  params['OUTPUTS_IDS_DATASET'],\n",
        "    'n_parallel_loaders': 1,\n",
        "    'maxlen': 50,\n",
        "    'normalize_probs': True,\n",
        "    'pos_unk': True and not is_transformer,\n",
        "    'heuristic': 0,\n",
        "    'state_below_maxlen': -1,\n",
        "    'predict_on_sets': ['test'],\n",
        "    'verbose': 0,\n",
        "    'attend_on_output': is_transformer\n",
        "  }\n",
        "predictions = nmt_model.predictBeamSearchNet(dataset, params_prediction)['test']"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "[30/07/2021 09:24:03] <<< Predicting outputs of test set >>>\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/numpy/core/_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            " Total cost: 9953.852238 \t Average cost: 3.322381\n",
            "The sampling took: 200.330725 secs (Speed: 0.066866 sec/sample)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2JcEpFJDTDs"
      },
      "source": [
        "Up to now, in the variable 'predictions', we have the indices of the words of the hypotheses. We must decode them into words. For doing this, we'll use the dictionary stored in the dataset object:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4EGTAOFXDYLX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1a45753-9812-4231-ebbf-7d93c46ee0e1"
      },
      "source": [
        "from keras_wrapper.utils import decode_predictions_beam_search\n",
        "vocab = dataset.vocabulary['target_text']['idx2words']\n",
        "samples = predictions['samples'] # Get word indices from the samples.\n",
        "\n",
        "predictions = decode_predictions_beam_search(samples,  \n",
        "                                             vocab,\n",
        "                                             verbose=params['VERBOSE'])"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:27:23] Decoding beam search prediction ...\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7MZVhj0IDd93"
      },
      "source": [
        "Finally, we store the hypotheses:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kznqPYZMDg8o",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6cd108bd-3b5e-459c-c958-eb930d89d65c"
      },
      "source": [
        "filepath = 'test.pred'\n",
        "from keras_wrapper.extra.read_write import list2file\n",
        "list2file(filepath, predictions)\n",
        "!head -n 4 test.pred"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "I would like to book a room until tomorrow , please .\n",
            "please wake us up tomorrow at a quarter past eight .\n",
            "I am leaving today in the afternoon .\n",
            "would you mind sending down our luggage to room number oh one oh , please ?\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jUntF5T6Dx2w"
      },
      "source": [
        "If we have the references of this split, we can also evaluate the performance of our system on it. First, we must add them to the dataset object:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-pccriZWDyqr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "addfd2d6-fab1-47fe-e18b-ec52fd04fe34"
      },
      "source": [
        "dataset.setOutput('examples/EuTrans/test.en',\n",
        "             'test',\n",
        "             type='text',\n",
        "             id='target_text',\n",
        "             pad_on_batch=True,\n",
        "             tokenization='tokenize_none',\n",
        "             sample_weights=True,\n",
        "             max_text_len=30,\n",
        "             max_words=0)\n",
        "keep_n_captions(dataset, repeat=1, n=1, set_names=['test'])"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:27:23] \tApplying tokenization function: \"tokenize_none\".\n",
            "[30/07/2021 09:27:23] Loaded \"test\" set outputs of data_type \"text\" with data_id \"target_text\" and length 2996.\n",
            "[30/07/2021 09:27:23] Keeping 1 captions per input on the test set.\n",
            "[30/07/2021 09:27:23] Samples reduced to 2996 in test set.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "riPUDl-xD1WM"
      },
      "source": [
        "Next, we call the evaluation system: the sacreBLEU package:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rfLzm4QBD2oj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a74448b1-721d-4d11-cc39-7724e536542c"
      },
      "source": [
        "\n",
        "from keras_wrapper.extra.evaluation import select\n",
        "metric = 'sacrebleu'\n",
        "# Apply sampling\n",
        "extra_vars = dict()\n",
        "extra_vars['tokenize_f'] = eval('dataset.' + 'tokenize_none')\n",
        "extra_vars['language'] = params['TRG_LAN']\n",
        "extra_vars['test'] = dict()\n",
        "extra_vars['test']['references'] = dataset.extra_variables['test']['target_text']\n",
        "metrics = select[metric](pred_list=predictions,\n",
        "                                          verbose=1,\n",
        "                                          extra_vars=extra_vars,\n",
        "                                          split='test')"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[30/07/2021 09:27:24] Computing SacreBleu scores on the test split...\n",
            "[30/07/2021 09:27:24] Bleu_4: 67.8701257426401\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jtfESZgUH6g_"
      },
      "source": [
        "And that's all!"
      ]
    }
  ]
}